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The findings from this review do not reflect the full body 
of research evidence on the Pathway Project. 


What is this study about? 

The study examined the impact of the Pathway 
Project intervention on students who were main- 
streamed Latino English language learners (ELLs). 

One hundred and three English teachers in 15 
schools in California were recruited and randomly 
assigned to either the Pathway Project condition 
or a comparison condition. 

The final analysis sample varied by outcome and 
included up to 50 teachers and 1 ,417 students in 
the Pathway Project condition and up to 51 teachers 
and 1,304 students in the comparison condition. 

The study assessed the effectiveness of the Pathway 
Project by comparing the English language develop- 
ment and general reading achievement of students 
in the intervention and comparison groups in the 
spring of the implementation year. 2 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: This study is a well-implemented 
randomized controlled trial. 


Features of the Pathway Project 


The Pathway Project is a professional development 
intervention that trains teachers to enhance the 
reading and writing abilities of mainstreamed ELLs. 

Students in the Pathway Project intervention first 
complete a pretest writing assessment, which is 
used by teachers to identify strengths and areas 
for growth. Based on their analysis of student 
writing samples, teachers develop lessons to 
address individual students’ needs. Teachers then 
use cognitive strategies (such as goal setting, 
tapping prior knowledge, asking questions, making 
predictions, and evaluating quality) to focus their text- 
based analytical writing instruction on these students. 


What did the study find? 

The study found, and the WWC confirmed, a sta- 
tistically significant positive effect of the Pathway 
Project intervention on student outcomes in the 
spring of the implementation year in the English 
language development domain. The average effect 
size calculated by the WWC for the English language 
development domain was 0.22. 

The study did not find a statistically significant or 
substantively important effect in the reading domain. 
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Appendix A: Study details 

Kim, J. S., Olson, C. B., Scarcella, R., Kramer, J., Pearson, M., van Dyk, D Land, R. E. (2011). A ran- 

domized experiment of a cognitive strategies approach to text-based analytical writing for main- 
streamed Latino English language learners in grades 6 to 12. Journal of Research on Educational 
Effectiveness, 4(3), 231-263. 


Setting 

The study was conducted in 15 secondary schools (nine middle schools and six high schools) 


in California’s Santa Ana Unified School District (SAUSD). Students in SAUSD are mainstreamed 
into regular English language arts classrooms when they score in the early advanced or 
advanced level on the California English Language Development Test (CELDT) or score in the 
intermediate or mid-basic level on the California Standards Test (CST) in English language arts. 

Study sample 

Students who were identified as English language learners (ELLs) and were eligible for main- 
streaming were randomly assigned to classrooms participating in the study. Within each grade 
and school, English teachers were randomly assigned to either the Pathway Project condition 
or the comparison condition. When a teacher taught more than one English class, the class 
with the highest percentage of students who scored at or above the intermediate level on the 
CELDT was selected to participate in the study. The Pathway Project materials were designed 
specifically for students at this level of English language proficiency. The final analysis sample 
varied by outcome and included up to 50 teachers and 1 ,41 7 students in the Pathway Project 
condition and up to 51 teachers and 1 ,304 students in the comparison condition. A random 
sample of students was selected to complete the Assessment of Literary Analysis (ALA) mea- 
sure, resulting in a total of 50 teachers (684 students) in the Pathway Project condition and 51 
teachers (709 students) in the comparison condition analysis sample. The groups were equiva- 
lent on the pretest ALA measure. A total of 95% of the students in the sample were Latino, 
88% were ELLs whose primary language spoken at home was Spanish, and 79% were eligible 
for free or reduced-price lunch. 

Intervention 

group 

Teachers in the intervention condition received training aimed at helping them enhance the 
reading and writing abilities of mainstreamed Latino ELLs through text-based, analytical 
instruction, using a cognitive strategies approach. This approach includes goal setting, tap- 
ping prior knowledge, asking questions, making predictions, and evaluating quality. Teachers 
then used these methods to help students make inferences and form interpretations after 
reading complex literary texts. Students completed a pretest writing assessment, which was 
used by teachers to identify strengths and areas for growth. Based on the teachers’ analysis 
of student writing samples, lessons were developed to address individual students’ needs. 

Comparison 

group 

Teachers in the comparison condition received professional development that emphasized 
interpreting test data and using it to improve CST scores, helping students improve their sum- 
marizing strategies during reading activities, forming professional learning communities, and 
understanding the core English language arts textbook. 
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Outcomes and 
measurement 


Support for 
implementation 


Reason for 
review 


The study assessed students’ English language development on the ALA and CST Writing 
subtest and reading achievement on the CST Reading subtest at both the pretest in October 
2007 and the posttest in May 2008. For a more detailed description of these outcome mea- 
sures, see Appendix B. 

Pathway Project teachers participated in 46 hours of training, including six full-day sessions (six 
hours each) and five after-school sessions (two hours each) distributed across the school year. 
Training was led by the developers of the Pathway Project. Experienced coaches also helped 
teachers incorporate cognitive strategies in reading and writing activities in their English lan- 
guage arts classrooms. Throughout the school year, teachers were provided with curriculum 
materials and a process for implementing the materials, including direct instruction, modeling, 
and guided practice. 

This study was identified for review by the WWC because it was supported by a grant to the 
University of California, Irvine (Principal Investigator: Carol Olson) from the National Center for 
Education Research (NCER) at the Institute of Education Sciences (IES). 
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Appendix B: Outcome measures for each domain 


English language development 

Assessment of Literary Analysis (ALA) 

This researcher-developed assessment tests the analytical writing skills of students in grades 6-12. After 
reading a short story, students are prompted to write an analytical essay. The essay is scored by two raters on 
a 6-point scale on six dimensions: quality and depth of interpretation, clarity, organization, appropriateness and 
adequacy of textual evidence, sentence variety, and language conventions. The correlation between first and 
second raters was 0.74. The rubric for scoring was based on the rubric used to evaluate the essay portion of 
the California High School Exit Examination, the California STAR 7 Direct Writing Assessment, and the National 
Assessment of Educational Progress. The ALA was administered at both pretest and posttest. 

California Standards Test (CST): Writing 
subtest 

The CST is designed to measure student mastery of English language arts in grades 6-11. It includes two writing 
subtests that measure written and oral English language conventions and writing strategies. The authors of this 
study created a writing subtest score based on the two writing portions of the CST. Reported reliabilities for the 
writing subtests range from 0.74 to 0.85 (Educational Testing Service, 2009). 3 

Reading 

CST: Reading subtest 

The CST also includes three reading subtests which measure word analysis, fluency, and systematic vocabulary 
development; reading comprehension with a focus on informational text; and literary response and analysis. The 
authors of this study created a reading subtest score based on the three reading portions of the CST. Reported 
reliabilities for the reading subtests range from 0.61 to 0.84 (Educational Testing Service, 2009). 3 


Note: The CST total score was also used in this study to measure student mastery of the English language arts content standards; however, this outcome is not included in this 
report. This total CST score is not eligible for review under the WWC’s English Language Learners topic area because it measures both reading and writing achievement, which fall 
in two separate domains. 
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Appendix C: Study findings for each domain 




Mean 







(standard deviation) 


WWC calculations 


Domain and Study 

Sample 

Intervention 

Comparison 

Mean 

Effect 

Improvement 


outcome measure sample 

size 

group 

group 

difference 

size 

index 

p-value 

English language development 

Assessment of Literary Grades 

101 

6.44 

5.82 

0.62 

0.35 

+14 

<0.01 

Analysis (ALA) 6-12 

teachers/ 

1,393 

students 

(1.56) 

(1.56) 





CST: Writing subtest Grades 

95 

0.04 

-0.05 

0.09 

0.09 

+3 

<0.05 

6-12 

teachers/ 

2,721 

students 

(0.97) 

(1.03) 





Domain average for English language development 




0.22 

+9 

Statistically 








significant 

Reading 

CST: Reading subtest Grades 

95 

0.02 

-0.02 

0.05 

0.05 

+2 

>0.05 

6-12 

teachers/ 

2,711 

students 

(0.97) 

(1.03) 





Domain average for reading 





0.05 

+2 

Not 








statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the 
comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an 
average student’s outcome that can be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average student's percentile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC; 
the study is characterized as having a statistically significant positive effect for the English language development domain because univariate statistical tests are reported for each 
outcome measure, the effect for at least one measure within the domain is positive and statistically significant, and no effects are negative and statistically significant. This study is 
characterized as having an indeterminate effect because the single effect reported is not statistically significant or substantively important. CST = California Standards Test. 

Study Notes: Baseline means for ALA and adjusted posttest means and standard deviations for the CST assessments were provided by the author in response to a request by the 
WWC. The WWC calculated the intervention group mean for ALA using a difference-in-differences approach (see the WWC Procedures and Standards Handbook, Appendix B) by 
adding the impact of the program (i.e., difference in mean gains between the intervention and comparison groups) to the unadjusted comparison group posttest mean. The post- 
test means for the CST Writing and Beading subtests were adjusted for the CSTtotal pretest score from 2007 by the authors of the original study, and the analysis was conducted 
on standardized outcomes (i.e., the standard deviation of the outcome measure was 1). The effect sizes for all three outcomes are based on a three-level hierarchical linear model, 
with students nested within classrooms within schools, by grade randomization blocks. A correction for multiple comparisons was needed for the English language development 
domain but did not affect significance levels. The p-values presented here were reported in the original study. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the English Language Learners review protocol, version 2.1 . The WWC rating applies 
only to the results that were eligible under this topic area and met WWC standards either with or without reservations, and not neces- 
sarily to all results presented in the study. 

2 One additional outcome was examined in this study, but is not included in this report because the measure assesses an outcome 
outside the scope of the English Language Learners review protocol: California Standards Test, English Language Arts Total Score. 

3 Educational Testing Service. (2009). California Standards Tests technical report, spring 2008 administration. Princeton, NJ: Author. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2012, August). WWC 
review of the report: A randomized experiment of a cognitive strategies approach to text-based analytical writing 
for mainstreamed Latino English language learners in grades 6 to 12. Retrieved from http://whatworks.ed.gov. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% Ip < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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